Local denoising applied to RAW images may outperform non-local patch-based methods applied to the camera output

State-of-the-art denoising methods achieve impressive results, even for large noise levels. However, they can not be implemented in camera hardware, mainly due to the fact that they are computationally too intensive. The aim of this paper is then to show that we can obtain comparable denoising results to the ones obtained with state-of-art methods by inserting a well-chosen fast denoising method at the right location in the camera processing pipeline. We evaluate our results visually and with respect to objective measures. Introduction Real-world scene camera acquisition generates noise due to physical and technological limitations, therefore denoising is performed at some stage of the formation of the output image. Nevertheless, the output image can still contain noise, especially if the photo is not taken with optimal camera parameters, or if the scene lighting conditions are challenging. Hence, there is still room for improvement in the denoising carried out in-camera in the image processing pipeline. Camera makers do not usually provide information about this pipeline, but some in-camera denoising techniques are well established, e.g. correlated double sampling, consisting in sampling two images, one with the shutter closed and another after exposure, and subtracting the latter from the former, thus reducing dark current noise; another common in-camera denoising method is coring, the thresholding of the DCT coefficients corresponding to high spatial frequency information that is usually associated with noise. What all denoising and any other in-camera processes must have in common is a low computational complexity, and a very good compromise between the processing power they require and the visual quality of the results they provide; we refer the reader to [3] for more details about the camera processing pipeline. On the other hand, over the last three decades, image denoising has been widely investigated and several approaches have been proposed. However, the methods developed mainly privilege the quality of the denoising, while ignoring the feasibility of their implementation in cameras. The first important class of denoising methods that appeared in the literature is the class of the so-called “local methods”, referring to the fact these methods modify pixels based on the values of their neighbors, either through a single shot procedure (e.g. convolution with a kernel) or through an iterative procedure (gradient descent of an energy). Local methods are simple and can be implemented in cameras. However, local methods do not separate well the noise from the edges and textures and the noise removal makes the edges and textures be oversmoothed. The best local methods are derived from the RudinOsher-Fatemi (ROF) model [13] based on the reduction of the Total Variation (TV) of an image. More recently, a breakthrough has been made simultaneously by the Non-Local Means (NLM) algorithm of Buades et al. [4] and the approach of Awate et al. [2] that perform denoising through the averaging of image patches all over the image domain. The efficiency of those “non local methods” to denoise natural images is due to the self-similarity of patches on natural images. Finally, the state-of-the-art denoising methods are derived from the Block-Matching and 3D Filtering (BM3D) algorithm [8] that combines patch-based approaches with frequency filtering in a non trivial way, outperforming the aforementioned patch-based methods in a great extent. We refer to [9] for a complete introduction to the denoising problem. Denoising can be applied at different stages in the camera processing pipeline. It can be applied on the RAW data, in which, however, the neighboring pixels are of different color; usually difficult for standard denoising algorithms. Alternatively, denoising can be applied on the monitor-ready image data. However, noise characteristics are extremely complex after all processing steps, see [14] for details. Due to the complex noise characteristics, the success of state-of-the-art denoising algorithms, tested on the usual image sets, can clearly drop on real camera data. In this paper, we therefore propose applying denoising at an earlier stage of the camera processing pipeline. State-of-the-art denoising methods, being based on patch comparisons in the image domain, are computationally too intensive for camera implementation. The aim of this paper is to propose a strategy to overcome this drawback. We show that applying a standard local method at some stage in the camera processing pipeline can provide better results than applying a nonlocal patch-based method to the output of the camera processing pipeline. More precisely, we show that, for noise levels that are not too large, the proposed strategy can produce more pleasant images and better results with respect to the PSNR and SSIM metrics. For a given camera, our strategy requires the simulation of its processing pipeline, as camera makers do not provide all the information on the pipelines their cameras are using. Inserting the denoising method at different stages of the (simulated) pipeline, and testing the quality of the corresponding camera output denoised images, we show that the best results are obtained when the denoising method is applied early in the pipeline on the RAW demosaicked image. The outline of the article is the following. In Section 2, we present the local and the non-local patch-based denoising methods we will use in our experiments. The proposed strategy as well as experiments on a standard digital camera are presented in Section 3. Finally, in Section 4, we discuss our current works: adding an extra step in the camera processing pipeline in order to improve our strategy and extend it to other types of cameras. Implementation of the denoising method inserted in the camera processing pipeline In the experiments we perform in Section 3, the local denoising method we insert in the camera processing pipeline is the ROF model. This model has been widely investigated both theoretically and numerically over the last two decades, and we refer the reader to [7] for a thorough analysis of that model. In what follows, we provide the necessary information for the understanding of our implementation of the model. The original formulation of the ROF denoising model is the following. We assume that the observed gray-level image I0 is the result of the corruption of a clean image Iclean with additive white Gaussian noise of standard deviation σ , i.e.